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(57) Abstract 

An eye gaze direction control arrangement using a light source (4) and a detector (5) giving the direction of eye gaze 
by detecting the positon on a user's eye (3) of the reflection of the light source and using such information to locate a cur- 
sor position on a video screen (2) and providing for a recalibration initiation achievable by a user to ensure consistent 
alignment of such a cursor with actual viewing direction. The calibration may be initiated by the user (1) providing multi- 
ple interruptions of the incident light within a predetermined time period (e.g. eye closures). The use (I) is then required to 
attempt to direct his gaze at various predetermined locations (7, 8, 9, 10) from which an average provides the calibration. 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international appli-^^^ 
cations under the PCT. , 



AT 


Austria 


GA 


Gabon 


MR 


Mauritania 


AU 


Australia 


GB 


United Kingdom 


MW 


Malawi 


BB 


Barbados 


HU 


Hungary 


NL 


Netherlands 


BE 


Belgium 


rr 


Italy 


NO 


Norway 


BG 


Bulgaria 


JP 


Japan 


RO 


Romania 


BR 


Brazil 


KP 


Democratic People's Republic 


SD 


Sudan 


CF 


Central African Republic 




of Korea 


SE 


Sweden 


CG 


Congo 


RR 


Republic of Korea 


SN 


Senegal 


CH 


Switzerland 


LI 


Liechtenstein 


SU 


Soviet Union 


CM 


Cameroon 


LK 


Sri Lanka 


TD 


Chad 


DE 


Germany, Federal Republic of 


LU 


Luxembourg 


TG 


Togo 


DK 


Denmark 


MC 


Monaco 


US 


United States of America 


n 


Finland 


MG 


Madagascar 






FR 


France 


ML 


Mali 







WO 86/01963 



PCT/AU85/00232 



IMPROVEMENTS RELATING TO AN APPARATUS AND METHOD RELATED TO CONTROL THROUGH 
EYE GAZE DIRECTION 

This invention relates to a method and apparatus 
associated with control through direction of eye 
gaze. 

The use of the direction in which a person 
5. is looking to effect control of some operation 
while not new of itself has hitherto incurred 
some difficulties. 

In order to ensure that there is an adequate 
correlation between the direction that a person 
10. is looking and a target location, it has been 

found that an act effecting a calibration provides 
significant advantage. 

As a major application of this method and 
apparatus will be to assist handicapped persons 
15. who may have difficulty keeping a head still, 

or even activate through limb pressure a controlling 
switch, it is a problem that this invention addresses 
to provide reliable means by which reasonable 
alignment can be maintained. 

20. The problem is to interpret whether a user 

has a temporary difficulty such as for instance- 
a tremor, or whether a major shift to relative 
positions has occurred so that original alignment 
characteristics no longer apply. 

25. An object of this invention accordingly 

is to provide means which will maintain through 
even difficult situations, good correspondence 
between a viewing direction of a user and a screen 
based cursor. 
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According to a preferred arrangement, there 
is a source of radiant energy providing a beam 
of light which when directed into a person's eye 
will result in a reflection and the position of this 
5. reflection with respect to the eye is responsive to 
the direction of gaze of that person. 

If a beam of radiant energy is shone into 
a person's eye, it is possible to identify the 
reflection of that radiant energy from an appropriate 

10c portion of the eye and by knowing the position 
and direction of the beam of radiant energy, it 
then becomes possible by determining the location 
of intersection onto a target which has a known 
relationship with respect to the source of the 

15. beam, some assumptions as to the gaze direction 
of the eye. 

Unfortunately, however, unless a person's 
head is fully restrained so as to maintain a certain 
selected eye position and the eye itself with 
20. respect to different people has idential characteris- 
tics, any assumptions made with respect to the 
eye direction have to be very approximate indeed. 

The purpose of this invention according 
to its broadest concept is to provide a technique 

25. for effecting a calibration procedure so as to 

provide for any proposed user and a system either 
a higher accuracy than has hitherto been available 
or a method to achieve such accuracy which has 
advantage over any system that has hitherto been 

30. used. 



In the application of the invention, especially 
in the preferred embodiment, the techniques involve 
both computers, hardware items of appropriate 
type, and software control of the hardware items. 

In broader references to the invention it 
is accordingly understood that such techniques 
can be achieved either by software controlling 
hardware, or by hardware in which the techniques 
are integral with respect to the hardware items. 

Accordingly the invention can be said to 
reside in an eye gaze direction control arrangement 
including - 

(1) a light source, 

(2) a detector adapted to detect the position 

of the reflection of the light source on the 
user's eye, 

(3) and means responsive to such detected 
position and adapted to control the 
position of a cursor on a screen and 
maintain this in an approximate alignment 
with a viewing direction of the user, 
said means being further adapted to 
effect a recalibration attempt sequence 
for calibration of the cursor viewing 
direction alignment upon an activation 
signal being effected by the user. 

It has been found according to a preferred 
technique that such an activation signal as can 
be effected by the user can include interrupting 
the reflected light pathway a selected number 
of times within a selected time period. 



This can simply mean the closing of the 
eyelids twice within a two-second period which 
would not be a normal closure and opening rate, 
but is something that a user can easily and quickly 
achieve even in most extreme circumstances. 

A turning away of the eye so that reflection 
communication is entirely lost, can also be used 
to achieve this result. 

For more able users, any limb movement can 
be used either to intersect a communicating beam 
or in another instance such movement can be used 
to activate a switch which can be interrogated 
from time to time in any controlling programme. 

When such a recalibration attempt sequence 
has been activated, it is a problem that a user 
may not momentarily be looking at the first calibra- 
tion viewing target location. 

Assistance can be achieved by using a second 
calibration viewing target location and then comparing 
correction factors that need to be applied in 
the case of one location as compared to the second, 
and if there is not a sufficiently close correlation, 
proceed to a further calibration viewing target 
location and comparing again a correction factor 
with previous results. 

Accordingly it may be .preferred that recalibra- 
tion is effected by a first calibration viewing 
target location to be caused to be noticeable 
by a user, and causing information to be stored 



WO 86/01963 



PCT/AU85/00232 



S. 

•identifying the angular direction then detected 
of a reflection of the radiant beam from the eye 
when it is to be expected to be viewing the calibration 
viewing target location then causing a second 
5, calibration viewing target location to be noticeable 
by the user, at a position different from the 
first said calibration position and such as to 
cause the eye of a user to assume a different 
angular relationship with respect to the detection 

10. equipment and causing information identifying 
such second angular position to be stored and 
then effecting a correction factor to be used 
for maintaining alignment of the position of the 
cursor on the screen with a viewing direction 

15. of the user. 

There is advantage if the calibration viewing 
targets are located in a non axial alignment such 
as along a horizontal axis, or in a further case 
along a vertical axis. 

20. In this way it is easier to compare results 

in which an x, y co-ordinate location should with 
a further calibration viewing target location 
on a common horizontal axis, have a very similar 
x co-ordinate reading, and likewise with locations 

25. on a vertical axis in which the co-ordinates will 
have a common or very close y co-ordinate reading. 

Some people have difficulty maintaining 
a consistent eye direction. This can be because 
of an essential instability in the eye or it may 
30. be a body tremor or it may be because of difficulties 
resulting from a disease such as, Parkinsons disease. 
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b. 

While an accurate correlation between eye 
direction and cursor position on a screen is desirable, X 
a greater tolerance of position can be accommodated 
with still the apparatus being useful, and accordingly 
5. for each recalibration process there is proposed 
that the said means are adapted to compare the 
correction factor calculated for the first of 
the target locations with a correction factor 
calculated for the second target locations and 

10. if the difference between the two is larger than 
a selected allowable difference, provide for a 
further target location to be offered to the user 
and continue the calculation of correction factor 
and comparison until at least two target locations 

15. provide a correction factor within the selected 
range . 

However, if the target locations needed 
to be offered exceeds a selected number, then 
there will be provided an enlargement of the range 
20. of acceptable difference in correction factor 
and a continuing offering of target locations. 

The method of this invention can be said 
to reside in the method of determining eye gaze 
direction control which includes the steps of 

25. directing a light source through a user's eye 

and detecting such reflection with means to enable 
the viewing direction of the eye to then be determin- 
able subject to calibration, and providing a calibra- s 
tion of a cursor viewing direction alignment upon 

30. an activating signal being effected by the user. 
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7. 

Reference has not been made thus far to 
the specific type of hardware that is being used 
as the target for viewing and while a video display 
unit can be used, at least according to the broader 
5. concept of this invention such calibration locations 
can be located in any environment which is useful 
for the purpose. 

The invention will be better understood 
when referred to a preferred embodiment which 
shall be described with the assistance of a drawing 
in which FIG. 1 is a schematic drawing showing 
a user arranged with a viewing screen and with 
a target light being directed through the eye 
of the user and being detected by a video type 
camera . 

Accordingly, the user 1 is positioned facing 
a video screen 2 with eyes 3 in approximate horizontal 
alignment. 

A small target light 4 radiates light onto the 
20. eyes 3 so that the position of the reflection of this 
light from the eyes is detectable within the video 
camera 5 . 

Such an arrangement is a known arrangement. 

However, to provide for consistent alignment 
of the apparent direction of sight of the eyes 
3 of the user 1, there is separately provided 
software within a computer which acting on previously 
provided factors such as the relative position 
and direction of the light source 4 and the camera 
5 and the position of the viewing screen 2, a 
correction factor which will modify each received 
target light reflection and cause a cursor to 



10. 



15. 



25. 



30. 
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be aligned on the video screen 2 in a perceived 
alignment with the viewing direction of the 
eye 3 . 

However, substantial difficulties in the 
5. location of the head 6 of the user 1 can alter 
the relative angular relationships of the light 
4, the eye 3 and the screen 2. 

Such a substantial realignment is best then 
followed by altering the correction factor so 
10. that viewing direction once again can be brought 
into alignment with a cursor and therefore the 
location of interest at any time on the viewing 
video screen 2. 

Accordingly, a calibration technique is 
IS. provided in which a first calibration viewing 

target location 7 is displayed on the video screen 
2 this comprising a cross which is rotated about 
its centre fast enough to be noticeable but not 
too fast as to be indistinguishable. Such a rotation 
20. speed can be two rotations each second. The programme 
provides for the first calibration viewing target 
location 7 to be maintained for a sufficient period 
of time for calculation of a new correction factor', 
and also to ensure consistent viewing direction 
25. during a first calibration period of time. 

Such a period for the first location 7 can 
be three seconds after which a second location 
at 8 is then used, after which a third location 
9 is used, after which a fourth location 10 is used. 

30. For each of the further locations, a further 

correction factor is calculated, such a correction 
factor being a discernible difference between 



a first standard x and y co-ordinate position 

of a cursor and the difference that would appear 

to be necessary to bring the cursor to a new position 

being the actual viewing impact position of the 

eye of the user 3. 

Subsequent to a second viewing target location 
being proposed at 8, and this being at an inclined 
relative relationship to the position of target 
7, there will then be two correction factors available 

The programme is adapted to compare the 
respective correction factors and if necessary 
compare any difference between the two correction 
factors to a preselected maximum tolerance. 

By using two targets which have an inclined 
relationship with respect to both x and y axes, 
a single correction factor can then be proposed 
taking into account both x and y deviations 
and also deviations in inclination. 

A further check is then available by performing 
two further calculations on the new locations 
9 and 10 and obtaining a proposed correction factor 
for once again both x and y axes and orientation, 
and then comparing such correction factor with 
that obtained with targets 7 and 8. 

If these are within a selected tolerance, 
preferably the average of the two correction factors 
can be calculated and placed in memory for then 
afterwards continuing application to each of the 
incoming detected reflected eye angular relationships. 

As stated previously, however, it is not 
always clear that a user has been looking precisely 
at the target location or within an acceptable 
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10. 

range for such a target location, and accordingly 
a further check is made comparing the y axis deviation * 
between perceived eye angles for target locations 
7 and 9 which are indeed on a common horizontal 
5. axis and 8 and 10 which are also on a common horizon- 
tal axis, and likewise the tolerance of x axis 
alignment for target locations 7 and 1 in one instance 
and 9 and 8 in the other. 

It is not always easy for a user 1 to call 
10. up a recalibration attempt sequence, and accordingly 
there is provided that if within any two seconds 
of time while the apparatus is in use and the 
eye of the user is having a target light reflection 
detected during other than a recalibration attempt 
15. sequence, the path of light is intersected twice 
for a period which lies within the range of 0.1 
to 0.5 of a second on each occasion, then this 
will activate a recalibration sequence in which 
the apparatus will revert from whatever other 
20. task it is operating to commence showing on the 

screen a recalibration target location and continuing 
with the sequence as has been previously described. 

The cutting off of reflection can be achieved 
in any of many different ways, but according to 
25. this embodiment this is achieved by the eyelids 

of the user being closed within the time constraint 
of two times within a two-second period. 

The apparatus will not be able to readily 
interpret whether the shut-off of reflection is 
30. by way of an eyelid or by way of a full turning 
of the head away from the viewing apparatus, but 
of course the result can be the same provided 
the same time restraint is achieved. 
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II . 

Likewise, a separately activated switch 
can be operated and can be discernible to call 
for a recalibration attempt sequence in a further 
embodiment . 

5 * In the case of some users, it will be difficult 

for them to ensure a sufficiently consistent eye 
direction for a sufficient period of time to enable 
adequate recalibration at any instance. 

The apparatus is intended to be of value 
10. to even the most highly handicapped persons and 

it is well understood that in some instances accuracy 
of alignment of eye direction is less important 
than continuing ability to control the position 
of a cursor and activities controlled by such 
15. position, even though such control within an adequate 
correction factor might be quite approximate. 

In other words, it could be that a person 
using the apparatus could be placed at risk if 
at least some control is not available even if 
20. generally inaccurate. 

Accordingly, there is proposed that before 
a recalibration attempt, a first comparison of 
correction factors is made and compared to a first 
selected tolerance. 

25# If the comparison is not within the selected 

tolerance, a further two inclined orientated target 
viewing locations will be provided and once again 
such correction factors as are calculated compared 
with an immediately preceding calculated correction 

30. factor. 
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Such sequence will be repeated four times 
after which the tolerance allowable will be enlarged 
by a preselected factor and a further recalibration 
attempt sequence will be effected. 

5e Such enlargement upon a selected number 

of unsuccessful recalibration attempt sequences 
will be continued until at least the best possible 
correction factor in the circumstances has been 
achieved. 

10- This then describes the embodiments presently 

put into practice. 

The appartus and techniques are applied 
by computer programmes - 

The techniques of computer programming are 
15. common place and the embodiment can be effected 
by appropriate programming of any appropriate 
computer with sufficient memory and with appropriate 
means to interpret results from a video camera, and 
control the positioning of a cursor and other 
20* illustrations on a video display screen. 

While reference has been made to an activation 
signal for a recalibration attempt sequence being 
achievable by the user comprising breaking the 
beam, a further signal can include detection of 
25. a selected eye movement of the user, or sequence 
of movements . 

If such . a sequence is not that which is 
likely to occur in normal usage, such activation 
can then be voluntarily effected. 
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13. 

In a preferred instance the movement of 
the eye to follow from a first target to a second 
target, which is horizontally coincident with 
the first target but spaced well apart from this, 
5. and then back again and have the detection for 

recalibration attempt sequence effected upon this 
movement being repeated at least three times within 
a three-second period can be appropriate • 

Obviously alternate sequences intended not 
10. to duplicate a sequence ordinarily occurring can 
be used. 
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TflE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS: 

1. An eye gaze direction control arrangement 
including - 

(1) a light source, 

(2) a detector adapted to detect the position 
5. of light from the light source being 

reflected from a user's eye, 

(3) and means responsive to such detected 
position and adapted to control the 
position of a cursor on a screen and 

10- maintain this in an approximate alignment 

with a viewing direction of the user, said 
means being further adapted to effect a 
recalibration attempt sequence for 
calibration of the cursor viewing 

15. direction alignment upon an activation 

signal being effected by the user. 

2. An eye gaze direction control arrangment 
as in Claim 1 further characterised in that the 
said means are adapted to effect a recalibration 
attempt sequence upon a selected number of interrup- 

5. tions of the reflected light within a selected 
period of time. 



3. An eye gaze direction control arrangement 
as in Claim 1 or 2 further characterised in that 
recalibration is effected by a first calibration 
viewing target location to be caused to be noticeable 
by a user, and causing information to be stored 
identifying the angular direction then detected 
of a reflection of the radiant beam from the eye 
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15. 



when it is to be expected to be viewing the calibra- 
tion viewing target location then causing a second 

10. calibration viewing target location to be noticeable 
by the user, at a position different from the 
first said calibration position and such as to 
cause the eye of a user to assume a different 
angular relationship with respect to the detection 

15. equipment and causing information identifying 
such second angular position to be stored and 
then effecting a correction factor to be used 
for maintaining alignment of the position of the 
cursor on the screen with a viewing direction 

20. of the user. 



4. An eye gaze direction control arrangement 
as in Claim 2 or 3 further characterised in that 
the said means are adapted to enter a recalibration 
attempt sequence upon a selected eye of the user 

5. being closed and opened a selected number of times 
within a selected period of time. 

5. An eye gaze direction control arrangement 

as in either of Claims 3 or 4 further characterised 
in that the said means are adapted to effect a 
recalibration attempt sequence upon an initial 
5. triggering by the user in which the sequence includes 
at least three calibration viewing targets, two 
of which are aligned in a vertical direction and 
two of which are aligned in a horizontal direction. 

6. An eye gaze direction control arrangement 
as in Claim 5 further characterised in. that the 
said means are adapted to compare horizontal and 
vertical alignment data from the calibration viewing 

5. targets and if the difference is larger than a 
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selected allowable difference, provide for a further 
target location to be offered to the user and continue 
the calculation of correction factor and comparison 
until at least three target locations provide a 
10. comparison within the selected range. 

7 . An eye gaze direction control arrangement 
as in Claim 6 further characterised in that the 
said means are adapted to enlarge the range of 
acceptable difference in horizontal and vertical 

5. alignment of the detected eye gaze direction when 
viewing the calibration target locations after 
more than a selected number of comparisons have not 
been within the selected range. 

8. A method of determining eye gaze direction 
control which includes the steps of directing 

a light source through a user's eye and detecting 
such reflection with means to enable the viewing 
5. direction of the eye to then be determinable subject 
to calibration, and providing a calibration of 
a cursor viewing direction alignment upon an activating 
signal being effected by the user* 

9. An eye gaze direction control arrangement 
substantially described in the specification with 
reference to and as illustrated by the accompanying 
drawings . 
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